Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce subassembly offset output artifact #15710

Draft
wants to merge 4 commits into
base: develop
Choose a base branch
from

Conversation

nikola-matic
Copy link
Collaborator

No description provided.

@nikola-matic nikola-matic force-pushed the introduce-subassembly-offset-output-artifact branch from d086ad9 to e4f8896 Compare January 17, 2025 10:39
Copy link
Member

@cameel cameel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some bugs, some missing parts and the overall implementation could be done more robustly.

I'm also not sure if the whole design actually accomplishes our goal. Only giving sub locations may not be enough to locate metadata without heuristics because data objects are not placed in separate subs at evmasm level.

See comments below for details.

@@ -1215,6 +1216,7 @@ LinkerObject const& Assembly::assembleLegacy() const
);
immutableReferencesBySub = linkerObject.immutableReferences;
}
subAssemblies.push_back(linkerObject);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The linker object is already available as m_assembledObject on each Assembly in m_subs. No need to create another copy.

size_t start;
size_t length;
bool isCreation;
std::string bytecode;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not the same thing as bytecode stored directly in LinkerObject. The name should contain enough information to make that clear:

Suggested change
std::string bytecode;
std::string linkedBytecodeHex;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, why do you even store it? It's relatively large (on the same order as the size of the whole LinkerObject) and all you ever do with it is check its size (which you could get from length instead).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the feature to the CLI too. We should keep them at parity (and it's a pain for testing/development if the only way to access a feature is through StandardJSON).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, Yul compilation is not covered. Aside from general inconsistency, this makes two-step compilation less powerful, which may be a problem for future parallelization.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be implemented for EOF as well (i.e. in assembleEOF(), or, if possible, just in assemble() covering both with the same code).

Since we're at the stage where EOF is passing semantic tests (and they will be enabled by default quite soon), we should start requiring all new features for work on EOF as well.

{
for (auto& subAssembly: _subAssemblies)
{
subAssembly.start = _currentBytecodeSize - subAssembly.length;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm either missing something or this assumes that all subassemblies overlap and extend to the end of their parent assembly. Does it even work with more than one subassembly? If you have two subassemblies of the same length then you will end up with the same start location for both. And even if they're of different lengths it will be wrong.

EDIT: Yeah, you even have a test showing this overlap (standard_subassembly_offsets/):

                                        {
                                            "isCreation": false,
                                            "length": 780,
                                            "start": 1007
                                        },
                                        {
                                            "isCreation": true,
                                            "length": 130,
                                            "start": 1657,
                                            "subs": [

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should also have some asserts here. At the very least that a subassembly does not stick outside of its parent assembly.

Comment on lines +13 to +28
"subAssemblyOffsets": {
"subs": [
{
"isCreation": true,
"length": 130,
"start": 0,
"subs": [
{
"isCreation": false,
"length": 104,
"start": 26
}
]
}
]
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, is Sourcify ok with getting only subassembly locations? I've always been thinking about metadata as a separate subassembly myself, because it's separate data Object in Yul, but looking at the PR I remembered that it's actually not the case at evmasm level. Metadata goes into Assembly::m_auxiliaryData and logically becomes a part of each assembly's bytecode, not a separate object. It's simply appended after all the subs and other data. I think that due to this it will still be necessary to use heuristics to fish out its location within the assembly, even knowing location of all subassemblies.

I think we may need to include the start and length of Assembly::m_auxiliaryData separately for each sub (when it's non-empty). For completeness we may want to simply list all the data chunks (that would actually have been nice to have for compiler debugging in some cases).

CC @kuzdogan.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, we should have some tests that include data objects between assemblies.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if we had the exact location of metadata, we could create a more robust, Boost-based test that would get the structure info and the bytecode and diff the CBOR bit. It's not easy to spot a problem just looking at the values in command-line tests and we're not even shown the bytecode.

Comment on lines +13 to +19
"subAssemblyOffsets": {
"subs": [
{
"isCreation": true,
"length": 27,
"start": 0,
"subs": [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The top-level assembly is not nested in anything. Seems weird to put its data under 'subs'. It should be at the top level. Similarly with subAssemblyOffsets - it should rather be assemblyOffsets since it covers the whole binary, including the top.

Finally, we might end up including more info there later so maybe assemblyStructure would be a more future-proof name?

Suggested change
"subAssemblyOffsets": {
"subs": [
{
"isCreation": true,
"length": 27,
"start": 0,
"subs": [
"assemblyStructure": {
"isCreation": true,
"length": 27,
"start": 0,
"subassemblies": [

I'd use that name for LinkerObject::SubAssembly too, i.e. LinkerObject::Structure, which would nicely mirror Object::Structure which is similar in the sense that it stores metadata about the hierarchy (but on Yul level).

Comment on lines +10 to +16
"outputSelection": {
"*": {
"Storage": [
"evm.bytecode.subAssemblyOffsets"
]
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're not testing evm.deployedBytecode.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, bad indentation. And I'd put the contract in a separate file.

Comment on lines +1531 to +1535
{
Json ret = Json::object();
ret["subs"] = formatSubAssemblyOffsets(compilerStack.object(contractName).subAssemblyData);
creationJSON["subAssemblyOffsets"] = std::move(ret);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYI, with nlohmann-json you can create JSON objects in place:

Suggested change
{
Json ret = Json::object();
ret["subs"] = formatSubAssemblyOffsets(compilerStack.object(contractName).subAssemblyData);
creationJSON["subAssemblyOffsets"] = std::move(ret);
}
creationJSON["subAssemblyOffsets"] = {
{"subs", formatSubAssemblyOffsets(compilerStack.object(contractName).subAssemblyData)}
};

We should generally use that style whenever possible, because the intermediate variables does not add anything of value, especially one with a horrible name like ret. And the reversed order always makes things harder to follow.

Comment on lines +12 to +20
"subAssemblyOffsets": {
"subs": [
{
"isCreation": false,
"length": 87,
"start": 0
}
]
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The input file has two assemblies but the output shows only one. Is this a bug in your feature or an instance of #15725 (because the nested assembly is unreferenced)?

"subAssemblyOffsets": {
"subs": [
{
"isCreation": false,
Copy link
Member

@cameel cameel Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, I wonder why this isn't true. Does exporting and reimporting asm JSON lose the creation status or is it just because of how the artificial input was crafted? If the status gets lost, this would be a bug (and should be reported).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, we also need some coverage for optimized compilation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants